Chained Tests

Shared Fixtures (page X) are commonly used to reduce the amount of per-test overhead for setting up the fixture. Sharing a fixture involves extra test programming effort because we need to create the fixture and have a way of discovering the fixture in each test. Regardless of how the fixture is accessed, it must be initialized (constructed) before it is used.

Chained Tests is a way to reuse the test fixture left over by one test and the Shared Fixture of a subsequent test.

How It Works

Chained Tests involves taking advantage of the objects created by the tests that run before us in the test suite. This is very similar to how a human tester tests a large number of test conditions in a single test by building up a complex test fixture through a series of actions with the outcome of each action first being verified. We can achieve a similar result with automated tests by building a set of Self-Checking Tests (see Goals of Test Automation on page X) that do not do any fixture setup but instead relying on the "leftovers" of the test(s) that run before them. Unlike Reuse Test for Fixture Setup (see Creation Method on page X), we don't actually call another Test Method (page X) from within out test; we just assume that it has been run and has left behind something we can use as a test fixture.

When To Use It

I don't recommend using Chained Tests as a primary strategy because it is simply the test smell Interacting Tests (see Erratic Test on page X) recast as a pattern. It is, however, a valid strategy for refactoring existing tests that are overly long and have many steps building on each other. Such tests will stop executing when the first assertion fails. We can refactor such tests into a set of Chained Tests fairly quickly because it doesn't require determining exactly what test fixture we need to build for each test. This may be the first step in evolving them into a set of Independent Tests (see Principles of Test Automation on page X).

Chained Tests helps prevent Fragile Test (page X) because it is a crude form of SUT API Encapsulation (see Test Utility Method on page X). Our test doesn't need to interact with the SUT to set up the fixture because we let another test that was already using that API set it up for us. Fragile Fixture (see Fragile Test) may be a problem; if one of the preceding tests is modified to create a different fixture, the depending test will probably fail. This is also true if some of the earlier tests fail or have errors; they may leave the Shared Fixture in a different state from what this test expects.

One of the key problems is non-determinism of the order in which xUnit executes tests in a test suite. Most members of the family make no guarantees about this order (TestNG is one exception I know about); tests could start to fail when a new version of xUnit is installed or maybe even when one of the Test Methods is renamed (if the xUnit implementation happens to sort the Testcase Objects (page X) by method name.)

Another problem is that each test is a Lonely Test (see Erratic Test) because it depends on the tests that precede it to set up the test fixture. If we run the test by itself it will likely fail since the test fixture is not set up for it. This means we won't be able to run just the one test when we are debugging failures it exposes.

We must be aware that depending on other tests to set up the test fixture will result in tests that are harder to understand because the test fixture will be invisible to the test reader. This is a classic case of Mystery Guest (see Obscure Test on page X). It can be at least partially mitigated through the use of appropriately named Finder Methods (see Test Utility Method) to access the objects in the Shared Fixture. It is less of a problem if all the Test Methods are on the same Testcase Class (page X) and are listed in the same order as they are executed.

Variation: Fixture Setup Testcase

If we need to set up a Shared Fixture and we cannot use any of the other techniques to set it up (e.g. Lazy Setup (page X), SuiteFixture Setup (page X) and Setup Decorator (page X)), we can arrange to have a Fixture Setup Testcase run as the first test in the test suite. This is simple to do if we are using Test Enumeration (page X); we just include the appropriate addTest method call in our Test Suite Factory (see Test Enumeration). This is a degenerate form of Chained Tests in that we are chaining a test suite behind a single Fixture Setup Testcase.

Implementation Notes

The two key challenges in implementing Chained Tests are:

• To get the tests in the test suite to run in the desired order.
• To access the fixture leftover by the previous test(s).

While a few members of the xUnit family provide an explicit mechansim for defining the order of tests, most members make no guarantees about this order. We can probably figure out what order it uses by running a few experiments. Most commonly, this is either the order the Test Methods appear in the file or in alphabetical order by Test Method name (in which case, easiest solution is to include a test sequence number in the test name.) In the worst case scenario, we could always revert to Test Method Enumeration (see Test Enumeration) to ensure the Testcase Objects are added to the test suite in the right order.

To be able to refer to the objects created by the previous tests, we need to use one of the fixture object access patterns. If the preceding tests are Test Methods on the same Testcase Class, it is sufficient for each test to store any object references that subsequent tests will use to access the fixture in a fixture holding class variable. (Note that fixture holding instance variables won't do because each test is running on a separate Testcase Object and therefore don't share instance variables.)

If our test depends on a Test Method on a different Testcase Class being run as a part of a Suite of Suites (see Test Suite Object on page X), neither of these solutions will work. Our best bet will be to use a Test Fixture Registry (see Test Helper on page X) as the means to store references to the objects used by the tests. A test database is a good example of one.

Obviously, we don't want the test we are depending on to clean up after itself because that would leave nothing to reuse as our test fixture. That makes Chained Tests incompatible with the Fresh Fixture (page X) approach.

Motivating Example

Here's an example of a incremental Tabular Test (see Parameterized Test on page X) provided by Clint Shank on his blog:

public class TabularTest extends TestCase {  
   private Order order = new Order();
   private static final double tolerance = 0.001;
  
   public void testGetTotal() {
      assertEquals("initial", 0.00, order.getTotal(), tolerance);
      testAddItemAndGetTotal("first", 1, 3.00, 3.00);
      testAddItemAndGetTotal("second",3, 5.00, 18.00);
      // etc.}
  
   private void testAddItemAndGetTotal( String msg, int lineItemQuantity,
                                        double lineItemPrice,
                                        double expectedTotal) {
      // setup
      LineItem item = new LineItem(   lineItemQuantity, lineItemPrice);
      // exercise SUT
      order.addItem(item);
      // verify total
      assertEquals(msg,expectedTotal,order.getTotal(),tolerance);
   }
}
Example IncrementalTabularTest embedded from java/com/xunitpatterns/misc/TabularTest.java

This test starts off by building an empty order, verifies the total is zero and then proceeds to add several items verifying the total after each item. The main issue with this test is that if one of the sub-tests fails, all subsequent sub-tests don't get run. For example, suppose we have a rounding error that makes the total after the second item incorrect; wouldn't we like to see if the fourth, fifth and six items are still correct?

Refactoring Notes

We can convert this test to a set of Chained Tests simply by breaking up the single Test Method into one Test Method per sub-test. One way to do this is to use a series of Extract Method[Fowler] refactorings to create the Test Methods. This will force us to use an Introduce Field[JetBrains] refactoring for any local variables before the first "extract". Once we have defined all the new Test Methods we simply deleted the original Test Method and let the Test Automation Framework (page X) call our new methods directly. (If I don't have a refactoring tool handy, no worries. I just end the Test Method after each sub-test and type in the signature of the next Test Method before the next sub-test. Then I move any Shared Fixture variables out of the first Test Method.)

We need to ensure they run in the same order; since JUnit seems to sort the Testcase Objects by method name, we can force them into the right order by including a sequence number in the Test Method name. Finally, we need to convert our Fresh Fixture into a Shared Fixture; we do this by changing our order field (instance variable) into a class variable (a static variable in Java) so that all the Testcase Objects use the same Order.

Example: Chained Tests

Here's the simple example turned into three separate tests:

   private static Order order = new Order();
   private static final double tolerance = 0.001;
  
   public void test_01_initialTotalShouldBeZero() {
      assertEquals("initial", 0.00, order.getTotal(), tolerance);
   }
  
   public void test_02_totalAfter1stItemShouldBeOnlyItemAmount(){
      testAddItemAndGetTotal( "first", 1, 3.00, 3.00);
   }
  
   public void test_03_totalAfter2ndItemShouldBeSumOfAmounts() {
      testAddItemAndGetTotal( "second",3, 5.00, 18.00);
   }
  
   private void testAddItemAndGetTotal( String msg, int lineItemQuantity,
                                        double lineItemPrice,
                                        double expectedTotal) {
      // create a line item
      LineItem item = new LineItem(lineItemQuantity, lineItemPrice);
      // add line item to order
      order.addItem(item);
      // verify total
      assertEquals(msg,expectedTotal,order.getTotal(),tolerance);
   }
Example ChainedTest embedded from java/com/xunitpatterns/misc/ChainedTest.java

Note how the Test Runner (page X) gives us a better overview of what is wrong and what is working:

Unfortunately, we will not be able to run any of the tests by themselves while we debug this problem (except for the very first test) because of the interdependencies between the tests; they are Lonely Tests.

Page generated at Wed Feb 09 16:39:38 +1100 2011

Copyright © 2003-2008 Gerard Meszaros all rights reserved